read metadata and cards from tar without extracting multi-dim arrays by lol782 · Pull Request #563 · NNPDF/eko

lol782 · 2026-08-31T12:53:19Z

closes #533

Implements the cheap card-reading we discussed in #533.

Approach (following your instructions )

Added from_raw to TheoryCard, OperatorCard, and Metadata (in runcards.py / metadata.py). Each takes raw yaml, applies the v1/v2 upgrades, then calls from_dict. This is the upgrade logic lifted out of the EKO.theory_card / EKO.operator_card properties.
Added load_meta_and_cards_from_tar(path) in runcards.py: opens the archive and pulls only metadata.yaml / theory.yaml / operator.yaml via tarfile.extractfile — the operators are never extracted. Order is metadata → theory → operator (operator upgrade needs the theory dict).
Refactored the two EKO properties to reuse from_raw, so folder-reading and tar-reading share one code path.

Points I'd like your view on

Metadata legacy upgrade. v1/v2.update_metadata needs the folder (paths), which the tar path doesn't have. So I kept that legacy 0.13/0.14 upgrade inside Metadata.load and made from_raw do the plain build. Consequence: reading a very old EKO straight from tar won't auto-upgrade its metadata. Fine for modern files — is that an acceptable boundary, or do you want it handled differently?
Function placement. Put load_meta_and_cards_from_tar in runcards.py as you suggested.
Test depth. Added a test that reads the cards from a real archive and checks types + a few values. Let me know if you want stronger assertions.

Existing io tests pass locally and below is the new one.

felixhekhorn

Remember to run pre-commit

lol782 · 2026-09-05T18:40:05Z

fix the required changes, Apologies for not remembering pre-commit.

Operator card bug — from_raw wasn't forwarding the theory dict to v1/v2.update_operator. Fixed (this was the empty-operator failure on legacy versions).
Metadata legacy upgrade — removed the unused paths argument from update_metadata in v1.py/v2.py, and moved the version patching into Metadata.from_raw (which was where the legacy case was failing). load now just reads the file and calls from_raw.
Tests — added a fixture with the legacy + current archives (v1-0.13.tar, v1-0.14.tar, v3.tar) so both test_legacy.py and the new test reuse it. The new function is now tested across all versions, not just current. Both pass.
Runcards — Changed the function logic to improve performance. Previously, it scanned the member list up to three times; now, the number of scans is hardcoded to three.

and also changed the name from load_meta_and_cards_from_tar → read_eko_cards and for test function name too.

felixhekhorn

Please mark conversation which you believe to have addressed as resolved, so we know more easily which problems still need to be addressed
A simple way for me (and everybody else 🙃 ) to see you are human and not a bot is by seeing you thinking. For example e2b6366 could have been 4 commits - doing in turn what you describe here. Making small self-contained commits is not easy and I openly admit I'm not doing it myself all the time, but I try - and this way you can see me thinking.
this will also help with writing more informative commit messages as "fix the required changes" is not very helpful in the long run

lol782 · 2026-09-08T07:57:26Z

wait you're still thinking i am a bot 🥲.

felixhekhorn · 2026-09-08T08:00:24Z

wait you're still thinking i am a bot 🥲.

nono, of course not - but I want to help you to avoid that impression

PS: and to teach you how to write better commits along the way 🙈

lol782 · 2026-09-11T17:11:49Z

Sorry for the mess! I forgot to run pre-commit before pushing those changes.

lol782 · 2026-09-11T17:31:12Z

Also added the type hints for the newly added functions ,
The only thing left is now is the 1st change ,can you please elaborate it if i guessed it you want me to make me merge the Metadata.from_raw and the Metadata.load😵‍💫?

felixhekhorn

last missing item: please add an entry to the Changelog

lol782 · 2026-09-14T17:00:12Z

You probably saw it 😅 here's what happened, so you know the story:
I got to the changelog and noticed mine differed from the one on GitHub. Turned out my master had never been updated since I forked, so I added an upstream remote and rebased onto it. That's when I learned rebasing rewrites the commit hashes, so a normal push was rejected ,I used --force-with-lease to update the branch, which I gather is the expected step after a rebase.

End result is the six separate commits above, one per change, plus the changelog entry. Learned a fair bit about rebase/upstream in the process 🙂. Happy to adjust anything,but I'd lean toward keeping them as separate commits since that matches what you asked for earlier. Let me know.

felixhekhorn

Sorry another minor thing 🙃

aaand I'm learning something myself by reading PEP673 more carefully and understanding that Self is not always correct, but sometimes you need the literal name (however, not here, if I understood correctly 🙈 )

felixhekhorn · 2026-09-15T07:56:55Z

+        return content
+
+    @classmethod
+    def from_raw(cls, raw: dict) -> "Metadata":


Suggested change

def from_raw(cls, raw: dict) -> "Metadata":

def from_raw(cls, raw: dict) -> Self:

we can use Self here since a) we are already requiring py3.11 and b) we return an instance of cls and not Metadata explicitly (see also PEP 673).

change here and also runcards.py

felixhekhorn · 2026-09-15T08:01:58Z

keeping them as separate commits since that matches what you asked for earlier.

Indeed, I prefer separate commits since this way you can see the problem evolve and you can discard or retain individual steps if necessary

felixhekhorn · 2026-09-15T08:10:14Z

In case this applies to you: note e5d0050

lol782 · 2026-09-15T16:38:23Z

In case this applies to you: note e5d0050

Sure, I used Claude , via its chat interface, throughout this PR to understand the code and the issue, discuss the design, draft and refine code, and help debug. I reviewed, ran, and tested all changes myself and take full responsibility for the final contribution 😊.

lol782 · 2026-09-15T16:43:29Z

By the way since its coming to close, I’d love to keep the momentum going! so can you suggest what i should do next ? 🫠

felixhekhorn added enhancement New feature or request output Output format and management labels Sep 3, 2026

felixhekhorn requested changes Sep 3, 2026

View reviewed changes

Comment thread src/eko/io/metadata.py Outdated

Comment thread src/eko/io/runcards.py Outdated

Comment thread tests/eko/io/test_struct.py Outdated

Comment thread src/eko/io/runcards.py Outdated

scarlehoff reviewed Sep 3, 2026

View reviewed changes

Comment thread src/eko/io/runcards.py

scarlehoff reviewed Sep 3, 2026

View reviewed changes

Comment thread src/eko/io/runcards.py Outdated

felixhekhorn requested changes Sep 8, 2026

View reviewed changes

Comment thread src/eko/io/metadata.py Outdated

Comment thread src/eko/io/runcards.py Outdated

Comment thread tests/conftest.py Outdated

lol782 force-pushed the fix/533-read-card-from-tar branch from 07e0266 to 9196f58 Compare September 11, 2026 17:09

felixhekhorn requested changes Sep 14, 2026

View reviewed changes

Comment thread src/eko/io/metadata.py Outdated

lol782 added 6 commits September 14, 2026 21:40

read metadata and cards from tar without extracting multi-dim arrays

02a1be2

fix the required changes

cc3c080

rename legacy fixture and included EKO archive paths

f36faea

added the type hints to the new functions

e2f1ca5

refactor: reuse metadata parsing for legacy loading

05d85e0

documented EKO archive card loading in changelog

849b886

lol782 force-pushed the fix/533-read-card-from-tar branch from 0879755 to 849b886 Compare September 14, 2026 16:20

felixhekhorn requested changes Sep 15, 2026

View reviewed changes

update metadata and card loader type hints with Self

d406a90

	def from_raw(cls, raw: dict) -> "Metadata":
	def from_raw(cls, raw: dict) -> Self:

Uh oh!

Conversation

lol782 commented Aug 31, 2026 • edited by felixhekhorn Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Approach (following your instructions )

Points I'd like your view on

Uh oh!

felixhekhorn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lol782 commented Sep 5, 2026

Uh oh!

felixhekhorn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lol782 commented Sep 8, 2026

Uh oh!

felixhekhorn commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lol782 commented Sep 11, 2026

Uh oh!

lol782 commented Sep 11, 2026

Uh oh!

felixhekhorn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lol782 commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

felixhekhorn left a comment

Choose a reason for hiding this comment

Uh oh!

felixhekhorn Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

felixhekhorn commented Sep 15, 2026

Uh oh!

felixhekhorn commented Sep 15, 2026

Uh oh!

lol782 commented Sep 15, 2026

Uh oh!

lol782 commented Sep 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lol782 commented Aug 31, 2026 •

edited by felixhekhorn

Loading

felixhekhorn commented Sep 8, 2026 •

edited

Loading

lol782 commented Sep 14, 2026 •

edited

Loading